Automated protein crystallization imaging

ABSTRACT

An apparatus that automatically captures, stores and analyzes images of crystallization experiments contained in a number of crystallization plates. The apparatus includes a plate nest capable of accommodating protein crystallization plates of a plurality of different types, image acquisition optics, including an objective lens and an image capturing device, for focusing an image of a crystallization well, a light source including a bright field illumination device and a dark field illumination device, a nest positioning controller for moving the position of the plate nest with respect to the image acquisition optics to align various selected wells with said objective lens for imaging of the content of the wells. A database stores experiment information associated with each of the crystallization plates, the experiment information including identification of specific crystal forming parameter values, each of the crystallization plates is identified in said database by a unique identification code. A crystallization imaging controller controls crystallization imaging by retrieving the experiment information for each crystallization plate inserted into the apparatus, and controlling the nest positioning controller and the image acquisition optics in accordance with the retrieved experiment information. The apparatus captures multiple images of each crystal site using different light source and polarization conditions, and processes the multiple images to form extended fused images of each crystal site.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority under 35 U.S.C. § 119(e) ofprovisional application Ser. No. 60/371,115, filed Apr. 10, 2002, andincorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of proteincrystallization analysis and in particular to a process and apparatusfor capturing, indexing, analyzing and storing images of proteincrystallization samples.

Protein crystallization is a technique used to analyze the molecularstructure of proteins. In this technique, varying controlled ratios andcombinations of certain chemical reactants or catalysts are added to aprotein sample or “drop” to induce the protein to crystallize so thatits structure may be analyzed for use in biochemical researchapplications.

As shown in FIG. 1, one type of crystallization plate includes a numberof cells, each cell having a number of satellite wells (three in theexample). The cells are identified by row and column number, such as A1,B10, etc., and each satellite well is identified by its position numberin the cell (such as 1, 2 or 3). A quantity of a protein is placed intoeach of the satellite wells, and a different experimental amount of acrystallization catalyst is added to each satellite well in an attemptto induce crystallization of the protein. The mixture is given asufficient time to react, and then the satellite wells must be viewedunder a microscope to determine if sufficient crystallization hasoccurred. The particular amounts and/or ratios of crystallizationcatalyst material and protein placed in each satellite well must berecorded in order to determine which ratios and combinations aresuccessful in producing crystallizations. Typically, one protein will beused for each plate, and different catalysts and ratios of catalystswill be added to the protein drops in each satellite well.

Achieving successful protein crystallization is very time consuming andtedious, as only a very small number of crystallizations of a proteinoccur for many tens or hundreds of thousands of experimental reactantratios and combinations. Additionally, the contents of a well may bemore complex than a simple crystal structure. Specifically, the proteinsample (drop) may be turbid as opposed to clear, there may bemicro-crystal fields in the sample, protein may be aggregated, andcrystals may be embedded in a precipitate field. Since each of thesepossibilities requires specialized lighting conditions in order to bedetected, it is necessary to take many different pictures of the samewell under different lighting and focus conditions, thus adding evenmore complexity to the analysis process.

Accordingly, there is a need for improvement in obtaining, analyzing,and storing protein crystallization experiment images whereby allpotential information in an experiment well can be captured quickly andefficiently, and stored together with all experimental parameters andconditions associated with the well.

SUMMARY OF THE INVENTION

According to a preferred embodiment of the invention, an apparatus isprovided that automatically captures, stores and analyzes images ofcrystallization experiments contained in a plurality of crystallizationplates, the apparatus including a plate nest capable of accommodatingprotein crystallization plates of a plurality of different types, theplates each including a plurality of crystallization wells eachincluding a particular crystallization experiment, image acquisitionoptics, including an objective lens and an image capturing device, forfocusing an image of a crystallization well positioned under theobjective lens and electronically capturing the focused image, a lightsource including a bright field illumination device and a dark fieldillumination device, each of the bright field and dark fieldillumination devices being independently energized to introduce lightinto a crystal image well to enable capture of separate bright field anddark field images of the crystal image well by the image acquisitionoptics, a nest positioning controller for moving the position of theplate nest with respect to the image acquisition optics to align variousselected wells with the objective lens for imaging of the content of thewells, a database for storing experiment information associated witheach of the plurality of crystallization plates, the experimentinformation including identification of specific crystal formingparameter values, and wherein each of the plurality of crystallizationplates is identified in the database by a unique identification code,and a crystallization imaging controller for controlling crystallizationimaging by retrieving the information for each crystallization plateinserted into the apparatus, and controlling the nest positioningcontroller and the image acquisition optics in accordance with theretrieved experiment information.

According to another aspect of the invention, the imager apparatusobtains and stores multiple images of each desired crystallization wellunder a number of different lighting and/or focusing conditions, andprocesses the multiple images to form a fused image containing allavailable information pertaining to a crystal image well.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a protein crystallization plate for use in animager according to the present invention;

FIG. 2 is a diagram of a combination imaging light source according toone embodiment of the invention using LEDs;

FIG. 3 is a diagram of a combination imaging light source according toanother embodiment of the invention using an external halogen lightsource;

FIG. 4 is a diagram showing the use of polarization filters incombination with a light source according to another aspect of theinvention;

FIG. 5 a shows a series of narrow focus images of a protein crystaltaken at different focus depths, and

FIG. 5 b shows a composite extended focus image obtained by combiningthe narrow focus images of FIG. 5 a, according to another aspect of theinvention;

FIG. 6 is a flow diagram of one embodiment of a process for obtainingthe merged extended focus image of FIG. 5 b according to the presentinvention;

FIG. 7 is a flow diagram of one embodiment of a process for detection ofprotein drops on a “hanging drop” plate according to the presentinvention;

FIG. 8 is a perspective view of an imager housing showing a plateinserted into a plate nest holder according to one embodiment of theinvention;

FIG. 9 is a perspective view of the plate nest holder showing cut outsto accommodate a root plate gripper;

FIG. 10 is a perspective view of the imager light source configurationsaccording to one embodiment of the invention;

FIGS. 11 and 12 are respective views of a light polarizer filter in adisengaged position and engaged position;

FIG. 13 is a perspective view of the imaging optics components accordingto one embodiment of the invention;

FIGS. 14 and 15 are perspective views showing accommodation of differentimaging plate types by the plate nesting holder according to oneembodiment of the invention;

FIG. 16 is a perspective view showing an XY servo stage connected to theplate nest to enable precise well positioning control with respect tothe imaging optics;

FIG. 17 is a perspective view showing plate accommodation details of aplate nesting holder according to one embodiment of the invention;

FIG. 18 is a perspective view showing plate centering cams integratedinto the plate nesting holder according to one embodiment of theinvention;

FIGS. 19 and 20 are views showing the operation of the plate centeringcams to automatically center a misaligned portrait oriented plate in thenesting holder;

FIGS. 21 and 22 are views showing the operation of the plate centeringcams to automatically center a misaligned landscape oriented plate inthe nesting holder;

FIGS. 23 and 24 are views showing the operation of a Z-axis pressuresensor to detect pressure exerted on an imaging plate as it is engagedwith the imaging optics, to prevent damage to the imaging plate; and

FIG. 25 is a screen shot of a viewing and scoring computer applicationaccording to one embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention will now be described in conjunction with theaccompanying drawings. However, it will be understood that the followingdetailed description is for purposes of providing an explanation anddisclosure of the concepts of the invention only, and is not intended todefine the scope of the invention, which is defined solely by theclaims.

The imager has a plate nest or holder, as shown in FIG. 8, which extendsto a load/unload position via a servo controller as shown, to permit animaging plate to be loaded into the nest for imaging, and to be removedfrom the nest after image processing is completed. The loading andunloading of the plates may be performed manually, or via robotics froma plate storage area. Each plate is labeled with a bar code labelcontaining a unique identification code, which allows automaticidentification of each plate through scanning of the bar code label by abar code scanner. After a plate is loaded, the nest is retracted intothe imager for image processing.

The imager has the ability to be run in one of three modes: automatic,semi-automatic, or manual. In automatic mode, each plate is imagedpredicated on the image mask parameters selected by the user on a GUI(graphical user interface, see FIG. 25) during a plate setup phase. Animage mask is the marking of the wells from which images are desired.

In the example plate shown in FIG. 1, cells A1, B10, D4 and F9 aremarked by the user to be imaged. Each cell of the plate contains threesatellite wells. The satellite or sitting drop well is selected based onits position (1, 2, and/or 3 in this example). Depending on whichsatellite well setting is desired, the imager will automatically acquireimages for the selected wells per the preselected satellite wellselection. Once all images have been captured the nest is extended tothe load/unload position located outside of the main housing, as shownin FIG. 8.

In a semi-automatic mode, each plate is associated with an image mask asin the automatic mode, but the operator is allowed to make adjustmentsto the image acquisition process such as contrast, magnification, focus,light configuration, etc., for each image position. After the plate hasbeen positioned to acquire an image, a real-time image is displayed onthe operator's interface as shown in FIG. 25 and the operator may adjustthe settings as desired. Once the operator is satisfied with the imagesettings, the image is captured and the operator then can either acquireadditional images of the same well, or may choose to move to the nextimage location that has been preselected in the mask settings. Once allimage sites per the mask have been processed, the plate nest is extendedto the load/unload position where the plate is removed.

In the manual mode, the plate is loaded manually, and there is no imagemask associated with it. The operator manually selects the desiredsatellite wells or positions to image and is able to set the same imagerconfiguration settings as those mentioned in the semi-automatic mode.The operator controls plate loading and unloading before and after imageprocessing is performed.

The imager accommodates all presently known protein crystallizationplate formats. The imager plate nest supports all SBS (Society forBiomolecular Screening) format plates with heights ranging from thethinnest glass slide capable of supporting drops (approximately 0.1inch) to 1.6 inches as shown in FIG. 15. The nest also supportslarge-well Linbro/VDX plate sizes as shown in FIG. 14.

The imager stage, shown in FIG. 16, is a three axis servo-controlleddevice which allows motion in the plate horizontal plane (X and Y axes)and in the camera vertical plane (Z axis). Control of each axis isindependent of the other axes, allowing an infinite range of positions,which is necessary to accommodate all of the various plate formats thatare commercially available. The stage positioning resolution allows thesmallest well format to be accurately positioned for imaging. It hasbeen found that by linking the camera auto focus routine to the Z axisvia a mathematical algorithm focus compensation during zoom operationsis accelerated. In addition to the plate nest stage, the variabletransmitting polarizer and the lens focus mechanism also are servocontrolled.

The imager nest is shown in detail in FIG. 17. The nest is designed tohandle SBS and large-well Linbro formatted plates as explained above. Asshown, the nest is designed to have a relief to allow the maximum amountof light to pass through the wells located on the edge of the plate. Theplates are held in the nest by a number of nest fingers as shown, suchthat the plates “ride” above the nest platform. It has been found thatthe relieved nest mitigates shadowing effects on the wells located nearthe edge of the plate, thereby dramatically improving image quality.

As shown in FIG. 15, the nest includes an active plate orienting system.This system uses a multiple cam mechanism which forces the plate intothe correct orientation regardless of plate placement in the nest. Asshown in FIGS. 19 and 20, the camming mechanism automatically centersplates inserted in a portrait orientation, and as shown in FIGS. 21 and22, the camming mechanism also automatically centers plates insertedinto the nest in a landscape orientation. The cam locating pins areopened as the plate nest is extended into the load position and are thenclosed when the nest is retracted into the main housing of the imager.Correct plate orientation is necessary to provide the required accuracyto locate and image the plate at high magnifications. The active cammechanism provides an automated methodology to correctly orient allplate types.

In a manual loading mode, the imager nest is fully extended with the camcentering devices open. A plate is placed in the nest either in portraitand landscape fashion depending on the plate type by hand. The operatorinstructs the imager via the GUI to begin imaging. Once the imageoperation is completed for that plate the nest is extended so that theplate is presented to the operator as shown in FIG. 8 with the platecentering cams open. The operator removes the plate by hand completingthe manual loading and unloading process.

In the automatic and semi-automatic modes, the imager may be loaded viarobotics. The nest is commanded to extend via a series of communicationsfrom a high level robot controller. Once extended the imager reportsback to the controller that it is ready to load and the robot thenplaces the plate in the imager in either portrait or landscapeorientation depending on the plate type. As shown in FIG. 9, the nesthas cut outs specifically located to allow the robot grippers to movedown into the nest while still gripping the plate. The plate is placedin the nest and the grippers are opened. The robot retracts and sends asignal to the imager that it is clear of the nest and that the plate islocated in the nest. The imager retracts the nest, which causes theplate centering cams to engage the plate and center the plate. Once theplate imaging operation has been completed, the nest is extended and theimager signals the robot controller that the plate is ready forretrieval. The plate cams are opened as the plate is extended. The robotretrieves the plate in the same manner as it loaded it by dropping intothe nest along the relieved cut-out portions.

The imager has the ability to acquire multiple high-resolution images ofeach well position using multiple various well lighting techniques. Themultiple image data is then fused to obtain a more complete and accurateunderstanding of the well contents. Thus, the imager is a truemulti-mode device that can analyze a well and determine more informationabout the well than is possible from a single image event. Specificallythe imager can determine crystal presence, clear versus turbid dropcontents, micro-crystal field, crystals embedded in a precipitate field,and protein aggregation. Using bright field backlighting crystaldetection is enhanced, while dark field lighting enhances precipitatedetection, and polarized light can be used to enhance micro-crystaldetection. Additionally, due to the properties of the various fluidsused in protein crystallization, the drops in sitting well experimentscan form a convex or concave shape, which diverts light either away ortoward the imaging microscope objective lens, causing undesirableshadowing effects. The dark field light mitigates these shadowingeffects by changing the incidence angle of the light on the well. Thedark field light will illuminate areas of the well that were notilluminated by the bright field light due to this shadowing effect. Thusthe combination of these two lights will mitigate the shadowing effects.The imager also provides for the means to oversize the backlight andthus floods the imaging area with light of various vectors, which hasbeen found to also mitigate this shadowing effect. It has been foundthat no one single image can fully represent the artifacts of a proteincrystallization drop in all cases and therefore a multimode imageprocess is required to produce accurate results in optical proteincrystallization analysis.

As shown in FIG. 10, the imager uses three distinct lighting techniques:bright field illumination, dark field illumination, and variable anglepolarized light illumination. Each light source has a distinct purposeas discussed above. FIG. 10 shows the orientation of each lightingcomponent in the imager. Each lighting technique is controlledseparately and can be energized independent of the other sources by aprocessing controller. This eliminates multiple light effects whencollecting multi-mode data. The imager housing enclosure shown in FIG. 8provides a light tight environment as well, which eliminates orsubstantially reduces undesired effects of ambient light entering theimaging plate wells.

The bright field illuminating technique is used as a backlight andilluminates the drop using a transmitted light approach. When employingthis technique the polarizing filter transmitter is disengaged as shownin FIG. 11, by rotating the polarizing filter so that a notch in thefilter aligns with the bright field light source. The light is a specialhigh-intensity low heat generation light. The heat load generated bytransmitted and reflected light sources is mitigated through varioustechniques, including the use of air cooling provided by a fan in theenclosure housing, as well as through the use of heat sink technologyincorporated into the light housing, and most importantly by controllingthe activation of light sources independently, thereby allowing thelight sources to be energized only when acquiring an image. Theindependent control of the light sources also provides for the abilityto vary intensity as well as to energize and de-energize each lightsource without impacting other light sources in the imager.

The bright field light or back light is built using LUMI-LED technologywhich offers longer lasting life, has the ability to be energized andde-energized many times without shortening the expected life and burnsvery cool as compared to a standard halogen source. The LUMI-LEDtechnology offers a great increase in intensity over standard LEDtechnology making the LUMI-LED a significant improvement over standardlighting techniques. It has been found that LUMI-LED technology providesa more stable back light source for protein crystallization imaging withimproved control and longer lasting life.

The dark field technique utilizes a standard dark field light source,which has been found to enhance certain aspects of a protein drop imageby changing the angle of incidence of the light with respect to theprotein crystal drop. The dark field source can be either an LED sourcewith independent control or a standard halogen source using a dark fieldadapter. The imager provides a means to use either type of lighting.

As shown in FIG. 2, the bright field and dark field light sources can beimplemented using LED or LUMI-LED technology, and controlled by adual-light controller via LED control cables. The LED dark field sourceprovides a stable long life illumination source for proteincrystallization with the added benefit of being able to independentlycontrol the light (e.g. vary intensity, energize and de-energize thesource without effecting other light sources). This allows thedark-field light to be de-energized during back light operations andenergized for dark field operations exclusively (i.e., the bright fieldlight is not energized). This arrangement allows for a clear anddistinct light combination to be used in the multi-mode approachprescribed by the imager, which is advantageous in producing higherquality protein crystal drop images.

Alternatively, as shown in FIG. 3 an external halogen light source canbe used in concert with a fiber optic dark field adapter. The benefit ofthis combination is the generation of very uniform and intense darkfield illumination without the addition of the halogen light heat sourceinternal to the imaging enclosure and the resulting negative impact tothe protein crystallization experiment, which requires a very uniformenvironment during the storage and time-lapsed imaging process. Theexternal halogen light source may remain on during all operationalmodes, including bright field and polarized light illumination.

In protein crystallography polarized light in concert with transmittedwhite light and a color camera can provide details that are not alwaysdiscernable to the human eye even with the aide of magnification.Because a property of most protein crystals is bi-refringence, polarizedlight can be used to find crystals that are either to small to bedistinguished by normal processing methods or crystals that may havebecome occluded in a precipitate field. As shown in FIG. 4 and also inFIGS. 11 and 12, a rotating transmitter polarizing filter is providedbetween the light source and the imaging plate, and an analyzerpolarizing filter is placed between the imaging plate and the imagingoptics.

The physics of light travel is such that if a polarizer transmitter andanalyzer are set to extinction angles that no light will pass;therefore, it has been found that if a protein crystal is not perfectlyaligned with either the transmitter or analyzer axes then the light rayswill be redirected by the presence of a protein crystal so that lightwill pass to the imaging optics with the analyzer and transmitter set atextinction, provided that the well is not bi-refringent. In thiscondition only crystals will be seen; however because many proteincrystal plates are composed of materials that during formation alsoexhibit bi-refringent properties a clean image is not returned duringimaging with the analyzer and transmitter at extinction. Instead theplate itself will redirect the light as it passes through the plate bychanging the angle of the light to a myriad of axes other than that ofthe transmitter. The result is a distinct diffuse light signature at theanalyzer. In order to mitigate these plate effects, it has been foundthat by varying the transmitter polarizing angles by rotation of thefilter and acquiring multiple images at these various angles and then bytransforming the negative of the image and performing image subtractionon the resulting set of images, the polarized composite result displaysa higher level of crystal confidence and shape than is possible with afixed polarizer transmitter/analyzer angle. The orientation of thefilters with respect to the object under inspection is critical and mustbe set so that the transmitter is on the light source side of the plateand the analyzer is on the objective or camera side of the plate asdepicted in FIG. 4 and also shown in FIG. 13.

In order to completely filter out light, a technique of employing twopolarizers set at 90 degrees with respect to each other is used. Thisfirst polarizer filter passes light in only one direction or axis withrespect to the angle of the polarizer, this filter is referred to as thetransmitter. Therefore with the imager setup shown in FIG. 12, when thetransmitter filter is engaged (i.e. inserted between the light sourceand the imaging plate), light is passed in one axis. This light passesthrough the crystal plate and then travels to the microscope objective.Prior to entering the objective the light passes through the analyzerpolarizing filter, which in the case of the imager doubles as a colorfilter. In order to accommodate the requirement of variable positionbased on user desires the transmitter is attached to a servomotor via anon-slip belt and pulley assembly. This arrangement provides completepositional control of the transmitter angle with respect to the proteindrop to be imaged. The filter is built by laser etching it into anoptically transparent disk and then notching out a portion of the diskso that it can be disengaged when bright field and dark field imagingare desired, as shown in FIGS. 11 and 12. The position-controlledtransmitter polarizing filter enhances image processing and datacollection in protein crystallization drop analysis.

As described above, a complete polarizing solution requires atransmitter polarizer and an analyzer polarizer. The imager accomplishesthis by utilizing the polarizing filter aspect of a color filter, whichis located between the microscope objective and the camera as shown inFIG. 13. It has been found that an LCD color filter has the correctoptical characteristics to act as an analyzing polarizer filter forprotein crystallization imaging. In addition to functioning as theanalyzer filter of the polarizing system, the color filter provides thecapability to acquire true color images without requiring a multiple CCDcamera. The color filter allows three images to be taken with a singleCCD array digital camera as shown in FIG. 13, and then combined to forma composite true color image. While other single CCD color cameraswithout the use of a color filter employ a mosaic approach known as theBayer Pattern to simulate a color image, the Bayer Pattern does notaccurately define a true color image because of the redundant greenpixel pattern used.

Turbidity Analysis

The imager supports the ability to conduct light scattering analysis tocalculate turbidity of the protein crystallization drop. The is achievedby measuring the amount of light collected over an extended time frameand then filtering the data set to determine light scattering patterns.It has been found that early stages, prior to crystal formation ofprotein aggregation can be detected using this methodology. The analysiscan be conducted using a variety of standard algorithms such as DiscreteFourier Transform, Fast Fourier Transform, Neural Net Filter, andStatistical Filters (such as LQE, LQR, LMS, etc.).

The turbidity detection methodology uses a light scattering orcollection technique to determine a turbidity measure of the well. Theimager can utilize different approaches to conducting this methodology.Two such methods are described below for completeness; however, bychanging the light source and collection device the imager can supportother approaches to turbidity analysis.

According to one embodiment, the imager can utilize a laser, which istuned to the appropriate wavelength and mimics the approximate size ofthe object under analysis (in the case of the imager, a protein crystaldrop). The laser beam is transmitted through the drop and a collector onthe opposing side of the drop measures the transmitted light level. Themore turbid the drop, the greater the resulting light attenuation. Thismeasurement is taken over a period of time and is coupled with the imagedata in order to show in clear to semi-clear drops early stages ofprotein aggregation as previously explained. The laser light passedthrough the crystal plate well is measured for intensity over a periodof time so that any deviation can be mapped over the sampling period andcorrelated with corresponding image data in order to detect early stagesof protein aggregation.

The imager also can employ the ability of a digital camera to measurelight intensity while employing bright field, dark field, or laserlighting techniques. The premise is identical to that stated above usinga laser emitter collector approach. Thus, according to an alternateembodiment, the light intensity focused through a well can be measuredover a period of time using the CCD array of the digital camera tocapture the light intensity variations.

As shown in FIG. 13, the imager uses a true microscope objective toenhance image depth of field (DOF) and provide the ability to focusthrough a very large range of field of views (FOV). The result of thistype of optics provides a large range of magnifications that can be usedby the imager. It has been found that this approach allows for moredynamic range of FOV, DOF and magnification than a standard set ofoptics with a variable magnification lens.

The imager employs a digital camera to capture a high resolution imageand to allow digital image processing to occur. The camera employs ahigh speed data transfer ability to move the data from the camera to theassociated user computer (not shown).

The camera has the capability to merge three single color images of thesame object into one composite true color image. The camera producesimages which can vary in resolution dependent on user desires. Thehigher the resolution of the image, the increased likelihood of smallobject detection; however, the image data size also increases withresolution. It has been found that lossy compressed data images can bemanually scored, but a lossless compression is required for automaticscoring accuracy. Therefore, according to a preferred embodiment, theimager saves a lossy version of the image for rapid manual scoring and alossless version for automated scoring in various image formats whichare user selectable.

The imager has the ability to acquire full color or monochrome images ata high resolution. The imager allows the operator to select the mode andresolution of the images to be taken in semi-auto mode or ahead of timefor the entire image set run in auto mode.

In FIG. 13 a Z Axis pressure sensor is shown mounted at the end of themicroscope objective. This sensor is used to detect the presence of thetop of a plate contained in the nest without applying a force to theplate. The sensor has a compliant ring that rests below the horizontalplane of the microscope objective (lens) as shown in FIG. 23. As thesensor comes in contact with a plate as shown in FIG. 21, the sensorring is pushed upward. As the sensor is displaced, a focused beam oflight to a optical detector that produces a signal when the light beamis obstructed. This detector sends a digital signal to the imager servocontrol system indicating an anomaly in the Z position of the lens. Thez-axis servo travel in the positive or down direction is then blocked toavoid a crash scenario, thus protecting the plate and the objective.

The imager also utilizes a vibration damped baseplate that isconstructed using a honeycomb composite. This plate design is lighterthan Granite damping plates and affords a ready methodology to mount theimager components to the baseplate without requiring special purposetools and fasteners. The vibration damped baseplate provides real timeplatform stability isolating stage movement during image acquisition ofprotein crystallization plates and therefore mitigates settling periodsrequired for image capture after plate movement is complete.

Software Components

The imager utilizes a sophisticated set of software components tocontrol, acquire, and analyze the protein crystallization imagingprocess. The imager control system reads specific instruction sets foreach plate via a database (not shown). The instructions are associatedwith the plate barcode so that each plate has a macro associated withthe experiment analysis set to be run for that particular plate. Thismethodology allows for unique experimental analysis steps to be followedon each plate independent of other plates in the system at that or anyprevious time. The following is an explanation of the various softwaremodules that are utilized in the imager.

The imager acquires images digitally and transfers the images from thecamera via a high speed data transfer medium such as IEEE 1394, FastSCSI, or a camera proprietary portal to an image processing PC. Theimages are associated with the plate barcode and well identificationcode (e.g. A1) in the database by having the control system identify theplate barcode either robotically or manually depending on the operationand then associating the well ID from the servo positioning softwarewith the acquired image or multiple images of each well. All of thisinformation is saved together with the actual image location in thedatabase. The imager uses an image mask to acquire the designated imagesfor a particular plate as described previously, which image mask also isstored in the database. The camera is triggered to acquire the imagefrom the imager control system software once the plate is correctlypositioned under the lens, which information is fed back to thecontroller from the servo positioning algorithms (which are well knownand therefore will not be further described herein). The images areeither concatenated in the digital camera before being transmitted tothe control system computer or are sent to the controller and areprocessed and merged into a high resolution composite image at thatpoint. The image processing and system control algorithms used by theimager are described below.

The imager has the ability to automatically merge multiple images ofnarrow depth of field as shown in FIG. 5A, into a single extended focusimage as shown in FIG. 5B. In this example four images of a protein dropwith crystal growth are merged into one extended focused image. Theprocess to create the merged image requires four steps, which aredescribed in FIG. 6. As each image is acquired, a median filter isapplied to a copy of the image to remove any noise in the image. Themedian-filtered image is then processed with a two dimensional edgedetector (Sobel detector) to create an edge image that is placed in alocal maximum map. After all desired images have been acquired andprocessed, each edge image is searched on a pixel-by-pixel basis withina variable neighborhood to determine which image has the greatest edgevalue. The image with the greatest edge value then contributes itsoriginal image pixel to the final merged image. The final merged imageis then constructed pixel-by-pixel using the process described above.

The imager also has the ability to automatically focus the imagecurrently in the view. This is a two step process. When the useractuates the auto focus capability the imager begins a search routineusing standard deviation of the image as a measure. A second order curvefit is used to predict the position corresponding to the maximumstandard deviation of the image. A second search is performed using asmaller range and step size and using a Tenengrad measure in place ofthe standard deviation. The maximum Tenengrad value is used as thefocused position. In addition the auto focus routine can use a region ofinterest to determine the optimum focused position allowing the user tofocus on a specific region of the image.

The imager further has the ability to automatically find the optimumexposure for the image. The camera has an integral capability to rapidlysample a range of exposure levels and select the value for the imagewith the lowest saturation level. By default auto exposure works withthe entire image but a region of interest can be specified. The imageralso has the ability to automatically set the white balance for theimage. The camera has an integral capability to sample a user selectedpixel in the image and adjust the individual color balance to force thatpixel to be white.

To determine whether polarization is to be used the imager controlsystem software reads the database using the plate barcode anddetermines the necessary experiment setting to analyze the plate. Oncethe plate is loaded in the imager the polarizer is positioned to thedesired angle and an image is acquired. Multiple images at variouspolarizer angles can be taken if the user has requested this in theexperiment setup. Each polarized image is saved and associated with asingle image event for the plate and the well in the database. Theseimages can be used to provide a more comprehensive view of the wellcontents based on the various polarization effects. As discussedpreviously most protein plates have a bi-refringent property andtherefore multiple polarization angles can provide more data to enhanceaccuracy.

The imager can save images in any format where a defined codec isavailable under the Windows Operating System. Such known codecs includeJPG, BMP, PNG, and TIF. Any other image encoding formats that providesimilar quality and capability also can be used according to theinvention. The imager may use uncompressed BMP format against which toconduct automated scoring (image processing) techniques. The userselects the format to be used for images to be saved for manual viewing.These formats may employ an image compression algorithm that willgreatly accelerate the annotation and manual viewing process.

For turbidity analysis, the imager associates the results with theplates and well positions in the database. The time lapsed results for asingle well are then passed through a filter which estimates andpredicts light scattering or absorption by the well and its contents.The algorithm takes the data points for a well for each image event andpasses the values into a statistical filter to calculate the deviationof the measured light intensity over the course of the recorded imageevents. In addition the values are statistical predicted for futureevents as well as predicted using a neural net estimator. These valuesare used to predict early stages of protein aggregation.

The data related to a particular well on an imaging plate is stored inthe database, and the stored data then is collated or fused by an enginewhich performs a series of statistical and rule-based processes to moreaccurately discern well contents and predict future well conditions.These results become the final score associated to the well in thedatabase while all of the original well data is still maintained. Byfusing the data sets associated with a well and applying basicstatistical methods, adaptive filtering techniques and custom rule-basedprocesses, a more accurate assessment of the well contents can be made.

The imager can use image processing algorithms to automatically findcrystals and precipitate in well images. These algorithms coupled withthe various images taken in the various different modes can be used tomore accurately characterize the well contents for proteincrystallization. The image processing algorithms utilize “blob” findingtechniques, morphological techniques, edge, and centroid detectiontechniques to fully qualify the well contents from a digital imageprocessing approach. By using standard advanced image processingsoftware algorithms clear drop discernment, precipitate recognition, andcrystal recognition can be achieved. Furthermore early stages of proteinaggregation can be modeled and identified using advanced statisticalmethods such as digital filter and adaptive filter methods. Suchstatistical and filtering algorithms are well known in the art andtherefore will not be further described herein.

The imager supports manual annotation and viewing of the images taken bythe system in one of two modes: offline post-image acquisition andonline real-time annotation during the acquisition cycle. FIG. 25depicts one of the interfaces supported by the manual image annotationsoftware. In this figure the image viewing area allows a detailed lookat the complete image as well as the ability to zoom to a region ofinterest either digitally (post image acquisition) or zoom and acquire anew image (real-time during the image acquisition process). The viewingwindow also supports measurement tools that are associated with rulersthat encompass the viewing window or by defining a region of interest onthe screen via a pointing device such as a mouse, using standardpractices adhering to the Windows Operating System Style Guide. Theimaging area can also be changed to view multiple images at one time ina thumbnail format. These images can be either time lapse views of asingle well on a particular plate or images from a common plate thatwere taken during a single image event (i.e., at the same relativetime). The location of each image in its perspective well is shown inthe plate map. Furthermore the plate map also shows all of the previousimage events for a particular plate across the top of the plate map inthe form of a tabs which show the date and time of the image event. Theprevious image events can be viewed in detail mode by selecting thedesired tab on this interface. The images can also be annotated via thisinterface in either offline or online modes. The annotation field shownin FIG. 25 is user configurable and is established from the database towhich the users have access in order to modify via another application.The annotation names and modifiers can therefore be customized by eachuser. The user can select the desired annotation and modifier toassociate with a particular image, and also has the option to describethe image in a narrative format. The resulting annotations areassociated with the image in the database via the plate barcode and welllocation fields.

The imager also supports the use of an automated scoring routine whichutilizes adjective annotation parameters defined in the database andassociates a numerical score to the appropriate adjective annotation.The scoring routine first assigns a set of numerical values to the fuseddata component of the image in question. These values are then relatedor mapped back to the adjective scoring system customized by the user.The adjective scoring system consists of a noun description and anadjective modifier (e.g., crystal/needles), which are related back tothe numerical score by a map maintained in the database. Thus theautomated scoring system can return numerical scores as well asassociated adjective scores.

The imager has the ability automatically find drops on hanging drop-typeplates. When a hanging drop image is acquired the steps shown in FIG. 7are performed to determine the bounding box of a drop. This allows theimager automation to zoom in and maximize the size of the drop in theimage. In particular, after an image is acquired, an Eigen process isperformed on it to generate an eigen image. The eigen image is maskedand edge-detected. The edge image is filtered through a median filter toremove noise and the filtered image is compared with the edge image todetermine a boundary of a drop in accordance with a predeterminedthreshold. The image in the determined boundary is then despeckled toform a bounding box drop image.

The imager database allows the tracking of all interactions of theimager with crystal plates. The first time a plate type is loaded intothe imager by a user, the user will define it for subsequent use byother plates of similar construction. Physical geometry information suchas number of wells along each axis, well pitch of each axis, number ofsatellite positions, offsets to satellite positions, plate height, andplate style (Linbro/SBS) is stored in the database. In addition, certaincamera settings related to exposure, gain, focus, and polarization areassociated with each satellite well defined on the plate type and storedin the database.

Once plate types are defined, the system can accept any plates of thatplate type for imaging. Plates are identified by a unique label orbarcode in the database. After loading and either scanning or entering aplate label or barcode, the imager will look up the plate in thedatabase. If it exists, it will load the plate type information thatthis plate is based on. Entry of new plates will cause the imager toprompt the user to select an existing plate type or define a new platetype. After the plate is associated with an existing plate type, theuser can begin imaging it. Each time that a user initiates the imagingprocess, a new image event record is created in the system. Image eventsgroup images of satellite positions on plates by the date they wereacquired. The database records the path to the satellite position, imageevent, image file path on disk, millimeters/pixel, polarization angle,image type (Bright Field, Dark Field, or Polarized), and image formats(JPG, BMP, PNG, TIF). Multiple images of the same or differing types maybe acquired for a particular satellite during the same image event. Onceimage acquisition is complete, the resulting images may be annotatedwith notes or scored via a set of descriptors and modifiers that eachcustomer defines. Annotations occur on a per image event satelliteposition basis—taking all images acquired during that event on thatsatellite position into account. The database allows multipledescriptors and modifiers to be selected for each annotation. Inaddition, multiple scoring methods are supported (manual, automatic,light scatter) by the database.

The user has the option to store either all images, non-clear images,images with crystal detection hits or any combination thereof. Thesettings for image storage are stored in the database and can befiltered by the user to obtain the desired results for that particularuser. The images themselves are not stored in the database, but ratheronly the image locations. This allows the database size to remain smalland manageable while still providing a link to the image via a singleinterface for future reference and possible scoring. The images for aplate for a particular event are located in the same directory that isnamed for the image event. The filepath is recorded and maintained inthe database. It has been found that maintaining the images external tothe protein crystallization database allows the database size to bemanaged to a smaller size increasing efficiency of search and queryalgorithms.

The imager also supports an automated plate scheduling algorithm. Theplate schedules are associated with the plate barcode in the database.The imager scans the database to register the next plate scheduled torun in the imager. The plate schedules can be superceded by manuallyintroducing a new plate to image at which time the plate scheduled torun will be delayed and run after the manual plate is completed. Thescheduling algorithm also supports multiple resource management whenmore than one imager resource is available to run a plate and multipleplates are vying for imager time. When more than one plate is scheduledto run on the imager at a single time the scheduling system supports aconflict resolution management system that arbitrates conflicts andreorders schedules dependent on user derived rules.

The imager has the ability to support all common format experiments usedin protein crystallization today. The results are all stored in a commondatabase regardless of the experiment type. The imager requires nodefinition of experiment type. The imager only requires plate type torun all experiment types. Various experiment types are handledautomatically without requiring user intervention to setup the imager toaccommodate each type. The following protein crystallization experimenttypes are supported:

Hanging Drop

These experiments can be conducted on a variety of plate type (Linbro,VDX, and Neuroprobe to a name a few). The imager allows for multipledrops per deep well and supports automated drop finding for fullyautomated image acquisition.

Sitting Drop

The imager will support any sitting drop plate on the market today withany well and satellite well configuration that can be made on an SBSformat plate. It can support very dense experiment plates 384 and 1536as well as large format 24, 48 and 96 well plates.

Microbatch

The imager will support all common microbatch plate setup formatswithout requiring any special purpose fixturing or user intervention toautomatically image.

Alternative Formats

The imager also can support sandwich drop experiment types and hasprovisions built into the nest to accommodate adapter plates necessaryto support future experiment types such as chip and micro slide basedexperiments.

Temperature Range (0° to 40° C.)

The imager can operate across a wide range of environmental conditionsthat are common to protein crystallization environments in labs todaysuch as fixed humidity, low temperature, and overpressure conditions.The temperature range supported by the imager is 0° to 40° C.

The invention having been thus described, it will be apparent to thoseskilled in the art that the same may be varied in many ways withoutdeparting from the spirit and scope of the invention. For example, whilethe preferred embodiment of the invention has been described inconjunction with protein crystallization experiments, the imager of theinvention can be used with any type of crystallization or precipitationexperiment wherein many different conditions must be configured in orderto discover optimal parameter values for crystal or precipitateformation. In the claims which follow, “crystallization” will refer tocrystallization or precipitation. Any and all such modifications aswould be apparent to one skilled in the art are intended to be includedwithin the scope of the following claims.

1. An apparatus that automatically captures stores and analyzes imagesof crystallization experiments contained in a plurality ofcrystallization plates, comprising: a plate nest capable ofaccommodating protein crystallization plates of a plurality of differenttypes, said plates each including a plurality of crystallization wellseach including a particular crystallization experiment; imageacquisition optics, including an objective lens and an image capturingdevice, for focusing an image of a crystallization well positioned undersaid objective lens and electronically capturing said focused image; alight source including a bright field illumination device and a darkfield illumination device, each of said bright field and dark fieldillumination devices being independently energized to introduce lightinto a crystal image well to enable capture of separate bright field anddark field images of said crystal image well by said image acquisitionoptics; a nest positioning controller for moving the position of saidplate nest with respect to said image acquisition optics to alignvarious selected wells with said objective lens for imaging of thecontent of said wells; a database for storing experiment informationassociated with each of said plurality of crystallization plates, saidexperiment information including identification of specific crystalforming parameter values, and wherein each of said plurality ofcrystallization plates is identified in said database by a uniqueidentification code; and a crystallization imaging controller forcontrolling crystallization imaging by retrieving said experimentinformation for each crystallization plate inserted into said apparatus,and controlling said nest positioning controller and said imageacquisition optics in accordance with said retrieved experimentinformation.
 2. An apparatus according to claim 1, wherein saidcrystallization experiments comprise protein crystallizationexperiments, and said crystallization imaging plates each contain aprotein samples combined with varying amounts of protein crystallizationinducing catalyst material.
 3. An apparatus according to claim 1,further comprising a transmitter polarization filter for polarizinglight from said light source to a crystal image well to provide apolarized image of said crystal image well for capture by said imageacquisition optics.
 4. An apparatus according to claim 1, whereinmultiple images of individual crystal image wells captured by said imageacquisition optics under varying lighting and/or focusing conditions areprocessed for forming a single fused image containing partial data fromat least two of said multiple images.
 5. An apparatus according to claim3, wherein said experiment information includes polarization angleinformation, and said controller controls the polarization angle of saidpolarization filter in accordance with said retrieved polarization angleinformation.
 6. An apparatus according to claim 5, wherein saidexperiment information includes multiple polarization angles for atleast one crystal image well, and said controller controls saidpolarization filter and said image acquisition optics to capturemultiple images of said crystal image well in accordance with saidretrieved multiple polarization angle information.
 7. An apparatusaccording to claim 1, wherein said apparatus analyzes light attenuationdata from said image acquisition optics for an image well obtained overa predetermined period of time to calculate a turbidity value for saidimage well.
 8. An apparatus according to claim 1, wherein said apparatusanalyzes light attenuation data from a turbidity detection lightdetector for an image well obtained over a predetermined period of timeto calculate a turbidity value for said image well.
 9. An apparatusaccording to claim 1, wherein crystallization plates are loaded andunloaded into said plate nest by a robotic plate loader/unloader.
 10. Anapparatus according to claim 1, wherein said plate nest furthercomprises an automatic plate centering cam mechanism that aligns a plateinserted into said plate nest into an appropriate alignment positionsubsequent to insertion of said plate into said plate nest.
 11. Anapparatus according to claim 10, wherein said automatic plate alignmentmechanism automatically aligns an inserted plate.
 12. An apparatusaccording to claim 1, wherein said image capture device comprises a CCDcamera, said optics further including a color filter for enabling truecolor images to be obtained using said CCD camera.
 13. An apparatusaccording to claim 3, wherein said image capture device comprises a CCDcamera, said optics further including a color filter for enabling truecolor images to be obtained using said CCD camera, and wherein saidcolor filter also functions as an analyzer polarization filter operatingin conjunction with said transmitter polarization filter to form apolarization system for allowing polarized light from saidcrystallization plate to be imaged.
 14. An apparatus according to claim1, wherein said nest positioning controller moves said plate nest in Xand Y axis directions in a plane perpendicular to said objective lens,and further moves said plate nest in a Z axis direction toward and awayfrom said objective lens, said apparatus further comprising a Z axispressure sensor for sensing the presence of a crystallization plateagainst said image acquisition optics and transmitting a signal to saidnest positioning controller to cause said nest positioning controller tostop movement of said plate nest toward said objective lens in said Zaxis upon detection of the presence of said crystallization plate. 15.An apparatus according to claim 1, wherein said crystallization platescomprise SBS type plates.
 16. An apparatus according to claim 1, whereinsaid crystallization plates comprise Linbro type plates.
 17. Anapparatus that automatically captures, stores and analyzes images ofcrystallization experiments contained in a plurality of crystallizationplates, comprising: a plate nest capable of accommodating proteincrystallization plates of a plurality of different types, said plateseach including a plurality of crystallization wells each including aparticular crystallization experiment; image acquisition optics,including an objective lens and an image capturing device, for focusingan image of a crystallization well positioned under said objective lensand electronically capturing said focused image; a light sourceincluding a bright field illumination device and a dark fieldillumination device, each of said bright field and dark fieldillumination devices being independently energized to introduce lightinto a crystal image well to enable capture of separate bright field anddark field images of said crystal image well by said image acquisitionoptics; a transmitter polarization filter located between said lightsource and said plate nest, for polarizing light from said light sourceat a desired polarization angle; said image acquisition optics furtherincluding an analyzer polarization filter operating in conjunction withsaid transmitter polarization filter as a polarization system forallowing polarized light from said crystallization plate to be imaged;wherein multiple images of individual crystal image wells captured bysaid image acquisition optics under varying lighting and/or focusingconditions are processed for forming a single fused image containingpartial data from at least two of said multiple images.
 18. An apparatusaccording to claim 1, wherein said bright field illumination devicecomprises a plurality of light-emitting diodes.
 19. An apparatusaccording to claim 1, wherein said dark field illumination devicecomprises a plurality of light-emitting diodes.
 20. An apparatusaccording to claim 17, wherein said bright field illumination devicecomprises a plurality of light-emitting diodes.
 21. An apparatusaccording to claim 17, wherein said dark field illumination devicecomprises a plurality of light-emitting diodes.
 22. A method ofanalyzing crystallization experiments contained in a crystallizationplate, comprising the steps of: illuminating a crystallizationexperiment well of a crystallization plate using a bright fieldillumination device to obtain a bright field illumination of said welland capturing an image of said well under bright field illumination;illuminating said crystallization experiment well using a dark fieldillumination device to obtain a dark field illumination of said well andcapturing an image of said well under dark field illumination; andprocessing said bright field image and said dark field image of saidwell to determine crystallization results of said experiment.
 23. Amethod as set forth in claim 22, further comprising the step ofilluminating said crystallization experiment well using polarized lightand capturing an image of said well under polarized light illumination,and wherein said processing step further comprises processing saidpolarized light image.
 24. A method as set forth in claim 22, whereinsaid processing step comprises the step of fusing said bright field anddark field images according to a data fusing algorithm.
 25. A method asset forth in claim 23, wherein said processing step comprises the stepof fusing said bright field, dark field, and polarized light imagesaccording to a data fusing algorithm.